Through advances in digital technologies combing multiple audio, video, and other kinds of pixel streams into a single transmission stream, conventional information media, that is means of communicating information to people such as newspapers, magazines, television, radio, and the telephone, can now be used for multimedia communication. “Multimedia” generally refers to text, graphics, audio, and video linked together in a single transmission stream, but conventional information media must first be digitized before the information can be handled in a multimedia format.
The estimated storage capacity needed to store the information carried by conventional information media when converted to digital data is only 1 or 2 bytes per character for text, but 64 kbits for one second of telephone quality audio, and 100 Mbits for one second of video at current television receiver quality. It is therefore not practical to handle these massive amounts of video, and other kinds of pixel streams into a single transmission stream, conventional information media, that is, means of communicating information to people such as newspapers, magazines, television, radio, and the telephone, can now be used for multimedia communication. “Multimedia” generally refers to text, graphics, audio, and video linked together in a single transmission stream, but conventional information media must first be digitized before the information can be handled in a multimedia format.
The estimated storage capacity needed to store the information carried by conventional information media when converted to digital data is only 1 or 2 bytes per character for text, but 64 kbits for one second of telephone quality audio, and 100 Mbits for one second of video at current television receiver quality. It is therefore not practical to handle these massive amounts of information in digital form on the above information media. For example, video telephony service is available over ISDN (Integrated Services Digital Network) lines with a transmission speed of 64 Kbps to 1.5 Mbps, but television camera grade video cannot be sent as is over ISDN lines.
Data compression therefore becomes essential. Video telephony service, for example, is implemented by using video compression techniques internationally standardized in ITU-T (International Telecommunication Union, Telecommunication Standardization Sector) Recommendations H.261 and H.263. Using the data compression methods defined in MPEG-1, video information can be recorded with audio on a conventional audio CD (Compact Disc).
The MPEG (Moving Picture Experts Group) is an international standard for digitally compressing moving picture signals (video). MPEG-1 enables compressing a video signal to 1.5 Mbps, that is, compressing the information in a television signal approximately 100:1. Furthermore, because the transmission speed for MPEG-1 video is limited to approximately 1.5 Mbps, MPEG-2, which was standardized to meet the demand for even higher picture quality, enables compressing a moving picture signal to 2 Mbps to 15 Mbps.
MPEG-4 with an even higher compression rate has also been standardized by the working group (ISO/IEC JTC1/SC29/WG11) that has advanced the standardization of MPEG-1 and MPEG-2. MPEG-4 not only enables low bit rate, high efficiency coding, it also introduces a powerful error resistance technology capable of reducing subjective image degradation even when transmission path errors occur. The ITU-T is also working on standardizing Recommendation H.26L as a next-generation picture coding method.
Unlike conventional video coding techniques, H.26L uses a coding distortion removal method accompanied by complex processing to remove coding distortion. Block unit coding methods using orthogonal transforms such as the DCT techniques widely used in video coding are known to be subject to a grid-like distortion known as block distortion at the coding block boundaries. Because image quality loss in low frequency components is more conspicuous than image quality loss in high frequency components, the low frequency components are coded more faithfully than the high frequency components in block unit coding. Furthermore, because natural images captured with a camera, for example, contain more low frequency components than high frequency components, the coding blocks contain more low frequency components than high frequency components. The coding blocks therefore tend to have substantially no high frequency components and adjacent pixels in a block tend to have substantially the same pixel value.
Furthermore, because coding is by block unit, there is no assurance that the pixel values will be substantially the same at the boundary between adjacent blocks, that is, that the pixel values will change continuously across the block boundary, even if the pixel values are substantially identical within each block. The result is that, as shown in FIG. 31 describing the concept of coding distortion removal, while the change in pixel values is smooth and continuous in the source image across the block boundary indicated by the dotted line as shown in FIG. 31(a), and the pixel values change continuously within each block as shown in FIG. 31(b) after the source image is coded by block unit, block distortion, that is, a discontinuity in pixel values only at the block boundary, occurs. Block distortion is thus a significant image quality problem resulting from image coding, but can be reduced by correcting the pixel values to be continuous across the block boundary as shown in FIG. 31(c). This process of reducing block distortion is called coding distortion removal (also referred to as “deblocking”).
When deblocking is applied at the video decoding stage, the deblocking filter can be used as a post filter as shown in the block diagram of a video decoder using a conventional decoding method in FIG. 32, or it can be used as an in-loop filter as shown in the block diagram of a video decoder using a conventional decoding method in FIG. 33. The configurations shown in these block diagrams are described below.
In the block diagram of a video decoder using a conventional decoding method shown in FIG. 32, a variable length decoder 52 variable length decodes encoded signal Str and outputs frequency code component DCoef. A de-zigzag scanning unit 54 rearranges the frequency components of the frequency code component DCoef in two-dimensional blocks, and outputs frequency component FCoef, the block unit frequency components. The reverse cosine transform unit 56 applies dequantization and reverse DCT operations to frequency component FCoef, and outputs difference image DifCoef.
Motion compensator 60 outputs the pixel at the position indicated by externally input motion vector MV from the reference image Ref accumulated in memory 64 as motion compensated image MCpel. Adder 58 adds difference image DifCoef and motion compensated image MCpel to output reconstructed image Coef. Deblocking filter 62 applies coding distortion removal to reconstructed image Coef, and outputs decoded image signal Vout. Reconstructed image Coef is stored in memory 64, and used as reference image Ref for the next image decoding.
The block diagram in FIG. 33 of a video decoder using a conventional decoding method is substantially identical to the block diagram of a video decoder shown in FIG. 32, but differs in the location of the deblocking filter 62. As will be known from FIG. 33 the decoded image signal Vout output from deblocking filter 62 is stored to memory 64.
The block diagram in FIG. 32 of a video decoder using a conventional decoding method shows the configuration and method used in MPEG-1, MPEG-2, MPEG-4, and H.263. The block diagram in FIG. 33 of a video decoder using a conventional decoding method shows the configuration and method used in H.261 and H.26L TM8.
With the block diagram in FIG. 32 of a video decoder using a conventional decoding method the reconstructed image Coef stored to memory 64 is not dependent upon the method applied by the deblocking filter 62. This allows developing and implementing various kinds of deblocking filters 62, including complex yet high performance filters as well as simple filters with relatively little effect according to the performance of the available hardware and the specific application. The advantage is that a deblocking filter 62 appropriate to the device can be used.
With the block diagram in FIG. 33 of a video decoder using a conventional decoding method the decoded image signal Vout stored to memory 64 is dependent upon the method employed by the deblocking filter 62. The problem here is that the filter cannot be changed to one appropriate to the hardware or application, but the advantage is that the same level of coding distortion removal can be assured in every device.
FIG. 34 is a block diagram of a coding distortion removal unit using the conventional coding distortion removal method. FIG. 34 shows the configuration of the deblocking filter 62 in FIG. 32 and FIG. 33 in detail. To efficiently remove only coding distortion from an image signal containing coding distortion, it is important to determine the amount and tendency for coding distortion in the image signal and then apply appropriate filtering so as to not degrade the actual image signal.
Because high frequency components account for much of the coding distortion, the general concept behind coding distortion removal is to survey the image signal to determine the ratio of high frequency components in the image signal, identify high frequency components in image signal pixels normally thought to not contain a high frequency component as coding distortion, and apply a high frequency component suppression filter to the coding distortion. This is possible because the correlation between adjacent pixels in an image signal is high, pixels containing a high frequency component are concentrated in edge areas, and dispersed high frequency components can be considered to be coding distortion.
This deblocking filter 62 was created by the inventors of the present invention based on content found in ITU-T Recommendation H.26L TML8.
Filtered pixel count controller 84 uses reconstructed image Coef to determine the pixel positions containing coding distortion, and outputs filtered pixel count FtrPel. Filter coefficient controller 86 uses filtered pixel count FtrPel and reconstructed image Coef to determine the filter coefficient (including the number of filter taps) appropriate to removing coding distortion from the indicated pixels, and outputs filter coefficient FtrTap. The filter processor 88 applies filtering to remove coding distortion from reconstructed image Coef using the filter coefficient indicated by filter coefficient FtrTap, and outputs decoded image signal Vout.
Disclosure of Invention
The conventional coding distortion removal methods described above are particularly effective at removing coding distortion, but the process is extremely complex and implementation difficult.
A further problem is that the amount of data processed per unit time is high.
Furthermore, no matter how effective the coding distortion removal method, it is impossible to accurately distinguish image signals and coding distortion without other additional information, and there is, therefore, the possibility that coding distortion removal will degrade image quality. This problem is particularly great with a configuration as shown in the block diagram in FIG. 33 of a video decoder using a conventional decoding method because the result of deblocking is used as the reference image and therefore affects the result of coding each subsequent picture.
An object of the present invention is therefore to provide a simple coding distortion removal method.
A further object is to provide a coding distortion removal method, a coding method, and a decoding method whereby the likelihood of degrading image signal quality can be reduced by applying high performance coding distortion removal with less possibility of degrading image signal quality as a result of removing coding distortion than the prior art.
To achieve this object, a coding distortion removal method according to the present invention for removing coding distortion from a picture uses different methods to remove coding distortion at boundaries where the motion compensation unit boundary matches the coding unit boundary match, and boundary depending on whether the boundary is a motion compensation block boundary or not, when the motion compensation block size is larger than the coding block size.
Because coding distortion at the boundary of the motion compensation unit differs qualitatively from coding distortion at the coding unit boundary, coding distortion can be efficiently removed from an image signal containing coding distortion by changing the filter used for deblocking according to the unit.
Furthermore, when coded motion compensation error is 0, coding distortion is preferably removed only at the motion compensation block boundary.
A further aspect of the invention is a coding distortion removal method for removing coding distortion from a picture by means of a step for extracting picture parameters from a picture containing coding distortion; a first step for identifying pixels for coding distortion removal using the picture parameters; a second step for identifying the method for coding distortion removal using the picture parameters; and a third step for removing coding distortion from the pixels identified by the first step using the coding distortion removal method identified by the second step.
By first computing picture parameters that can be used in both the first step identifying the pixels from which coding distortion is removed and the second step identifying the method used to remove the coding distortion, the operations performed in the first step and second step can be simplified by using these common picture parameters, and processing by the coding distortion removal method can be reduced without degrading image quality.
A further aspect of the invention is a coding distortion removal method for removing coding distortion from a picture whereby the pixels to be processed for coding distortion removal are identified by block based determination whether to remove coding distortion by block unit, and then pixel based determination whether to remove coding distortion for each pixel in the blocks determined to be removed by the block based determination.
By thus first determining by block unit whether coding distortion removal is needed, evaluation by pixel unit can be omitted in those blocks that do not need deblocking, and the processing performed by the coding distortion removal method can be reduced. Blocks that do not need deblocking (such as still image blocks where the pixels perfectly match the reference image) can be easily determined if the image coding information is used.
A yet further aspect of the invention is a coding distortion removal method for removing coding distortion in an area disposed on both sides of a block boundary between a first block and an adjacent second block in a picture having a plurality of blocks forming a moving picture image. This method has a comparison step for comparing a difference of pixel values of the first block and pixel values in pixels of the second block, and a parameter, corresponding to the average of a quantization parameter for the first block and a quantization parameter for the second block, for determining the method for removing coding distortion; and a removal step for removing coding distortion based on the result from the comparison step.
This enables the average of the quantization parameters for the adjacent blocks to be used when filtering both sides of the block boundary in a coding distortion removal process at the block boundary between different quantization parameters.
Another coding distortion removal method for removing coding distortion in an area disposed on both sides of a boundary line between a first block and an adjacent second block in a picture having a plurality of blocks forming a moving picture image has a decoding step for decoding a parameter for setting a threshold value when removing coding distortion; a comparison step for comparing a difference of pixel values in pixels of the first black and pixel values in pixels of the second block, and a specific threshold value based on the decoded parameter; and a removal step for switching the method for removing coding distortion based on the result from the comparison step.
Coding distortion can thus be efficiently removed from an image signal containing coding distortion by first superposing to each encoded signal a threshold value parameter used for coding distortion removal, and then prior to coding distortion removal detecting the threshold value appropriate to each encoded signal and using it to remove coding distortion.
Further preferably, the moving picture contains a slice composed of plural blocks; and the parameter is stored in slice header information in a code stream obtained by encoding image-data for the moving picture.
A further aspect of the invention is a moving picture coding apparatus for picture coding with reference to at least one of multiple reference images, wherein a plurality of coded images obtained by removing coding distortion using plural methods are the reference images.
By thus using plural images deblocked by at least two methods as reference images and sequentially selecting the appropriate one for reference, the picture obtained by efficiently removing coding distortion from an image signal containing coding distortion can be used as the reference image, and the compression rate of moving picture coding can be increased.
Further preferably the first method of the plural methods is a method that does not remove coding distortion in the coded picture, and the second method is a method that removes coding distortion in the coded picture.
A further aspect of the invention is a moving picture decoding apparatus for decoding with reference to at least one of multiple reference images, wherein a plurality of decoded images obtained by removing coding distortion using plural methods are the reference images.
By thus using plural images deblocked by at least two methods as reference images and sequentially selecting the appropriate one for reference, the picture obtained by efficiently removing coding distortion from an image signal containing coding distortion can be used as the reference image, and the coded signal can be corrected decoded.
Further preferably, the first method of the plural methods is a method that does not remove coding distortion in the decoded picture, and the second method is a method that removes coding distortion in the decoded picture.
A further aspect of the invention is a coding distortion removal method for removing coding distortion in an interlaced picture composed of odd-line pixels and even-line pixels. This method has an evaluation step for determining if a picture is a picture containing frame structure blocks having a specific number of odd-line pixels and a specific number of even-line pixels, a picture containing blocks of one field structure composed of a specific number of odd-line pixels, or a picture containing blocks of another field structure composed of a specific number of even-line pixels; and a removal step for removing coding distortion between adjacent frame structure blocks when the target block for coding distortion removal is a block in a picture in which all blocks are frame structure blocks, and removing coding distortion between adjacent field structure blocks when the target block for coding distortion removal is a block in a picture in which all blocks are field structure blocks.
Processing of the blocks for coding distortion removal can thus be changed based on whether the blocks are in a picture of frame structure blocks or a picture of field structure blocks.
Preferably, if the target block for coding distortion removal is a block of a picture containing frame structure blocks and field structure blocks, the coding distortion removal method also has a conversion step for converting a field structure block to a frame structure block; a comparison step for comparing a difference of pixel values in pixels of the field structure block and pixel values in pixels of the converted block with a specific threshold value; and a removal step for removing coding distortion based on the result from the comparison step.
In a further coding distortion removal method for removing coding distortion in an area disposed on both sides of a boundary line between a first block and an adjacent second block in a picture having a plurality of blocks forming a moving picture image, the first blocks are frame structure blocks having a specific number of odd-line pixels and a specific number of even-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels, and the second blocks are field structure blocks having one field composed of a specific number of odd-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels, and another field composed of a specific number of even-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels. The coding distortion removal method has a conversion step for converting a frame structure first block to a field structure block; a comparison step for comparing a difference of pixel values in pixels of the field structure second block and pixel values in pixels of the converted block with a specific threshold value; and a removal step for removing coding distortion based on the result from the comparison step.
When field structure blocks and frame structure blocks are adjacent, the target blocks for coding distortion removal can thus be adaptively processed.
Preferably, conversion from frame structure first blocks to field structure blocks switches by macroblock unit or units of two vertically adjacent macroblocks.
Further preferably, field structure second blocks are not converted to frame structure blocks.
In a further coding distortion removal method for removing coding distortion in an area disposed on both sides of a boundary line between a first block and an adjacent second block in a picture having a plurality of blocks forming a moving picture image, the first blocks are frame structure blocks having a specific number of odd-line pixels and a specific number of even-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels, and the second blocks are field structure blocks having one field composed of a specific number of odd-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels, and another field composed of a specific number of even-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels. The coding distortion removal method has an evaluation step for determining if the target block for coding distortion removal is a frame structure block or a field structure block; a conversion step for converting the frame structure first block to a field structure block when the target block is a field structure second block, and converting the field structure second block to a frame structure block when the target block is a frame structure first block; a comparison step for comparing pixel values in pixels of the target block with a specific threshold value; and a removal step for removing coding distortion based on the result from the comparison step.
When field structure blocks and frame structure blocks are adjacent, the target blocks for coding distortion removal can thus be adaptively processed.
Preferably, conversion in the conversion step from a frame structure block to a field structure block produces one field after conversion from odd-line pixels in the frame structure block, and produces the other field after conversion from even-line pixels in the frame structure block; and comparison of the difference and threshold value in the comparison step compares pixel values in pixels in one field of the second block and pixel values in pixels in one field of the first block after conversion, or compares pixel values in pixels of the other field in the second block and pixel values in pixels of the other field in the first block after conversion.
In a further coding distortion removal method for removing coding distortion in an area disposed on both sides of a boundary line between a first block and an adjacent second block in a picture having a plurality of blocks forming a moving picture image, the first blocks are frame structure blocks having a specific number of odd-line pixels and a specific number of even-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels, and the second blocks are field structure blocks having one field composed of a specific number of odd-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels, and another field composed of a specific number of even-line pixels in an interlaced picture composed of odd-line pixels and even-line pixels. The coding distortion removal method has a conversion step for converting a field structure second block to a frame structure block; a comparison step for comparing a difference of pixel values in pixels of the frame structure first block and pixel values in pixels of the converted block with a specific threshold value; and a removal step for removing coding distortion based on the result from the comparison step.
When field structure blocks and frame structure blocks are adjacent, the target blocks for coding distortion removal can thus be adaptively processed.
Further preferably, conversion from field structure second blocks to frame structure blocks switches by macroblock unit or units of two vertically adjacent macroblocks.
Yet further preferably, field structure second blocks are not converted to frame structure blocks.
Yet further preferably, conversion in the conversion step from field structure block to frame structure block produces a converted frame from pixels in a block of one field and pixels in a block of the other field, and compares pixel values in odd-line pixels in the first block with pixel values in odd-line pixels in the second block after conversion, or compares pixel values in even-line pixels in the first block with pixel values in even-line pixels in the second block after conversion.
Yet further preferably, the comparison step compares the difference and threshold value by groups of plural pixels aligned in line in a same direction as the boundary line at positions symmetrical to the boundary line.
This enables coding distortion to be removed in groups of plural pixels.
A yet further aspect of the present invention is a picture coding apparatus having a decoding unit for decoding a coded difference picture and outputting the difference picture; a motion compensation unit for outputting a motion compensated picture from a reference image; an adder for adding the difference picture and motion compensated picture, and outputting the merged picture; a coding distortion removal unit for removing coding distortion in the merged picture and outputting a reconstructed picture; and memory for storing the reconstructed picture as the reference image. The coding distortion removal unit removes coding distortion by means of any of the above-described methods of the invention.
A yet further aspect of the invention is a program for removing coding distortion from a picture by means of any of the above-described methods of the invention.
A yet further aspect of the invention is a program for picture coding using a decoding unit for decoding a coded difference picture and outputting the difference picture; a motion compensation unit for outputting a motion compensated picture from a reference image; an adder for adding the difference picture and motion compensated picture, and outputting the merged picture; a coding distortion removal unit for removing coding distortion in the merged picture and outputting a reconstructed picture; and memory for storing the reconstructed picture as the reference image. The coding distortion removal unit removes coding distortion by means of any of the above-described methods of the invention.
Other objects and attainments together with a fuller understanding of the invention will become apparent and appreciated by referring to the following description and claims taken in conjunction with the accompanying drawings.